Skip to main content

Reply to this post | See parent post | Go Back
View Post [edit]

Poster: glenn Date: Jul 19, 2004 1:31pm
Forum: web Subject: Re: Removal of a previous Web site?

you can also use a robots.txt file to specify which parts of your site are spidered by 'friendly' user agents... however, unfriendly user agents will ignore your robots.txt file...

see
http://www.robotstxt.org/wc/norobots.html
for more info on robots.txt files.